Skip to content

Refactor to_local_time() signature away from user_defined - #18707

Merged
Jefffrey merged 7 commits into
apache:mainfrom
Jefffrey:refactor-to-local-time-sig
Nov 26, 2025
Merged

Jefffrey merged 7 commits into
apache:mainfrom
Jefffrey:refactor-to-local-time-sig

Conversation

@Jefffrey

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #12725

Rationale for this change

Prefer to avoid user_defined for consistency in function definitions.

What changes are included in this PR?

Refactor signature of to_local_time away from user_defined.

Also some other refactors for to_local_time.

Are these changes tested?

Existing tests.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Nov 14, 2025
Comment on lines +115 to +118
signature: Signature::coercible(
vec![Coercion::new_exact(TypeSignatureClass::Timestamp)],
Volatility::Immutable,
),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signature change here

}
}

fn to_local_time(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refactored to_local_time to move it outside of ToLocalTimeFunc struct since it didn't really need the self


fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
match &arg_types[0] {
DataType::Null => Ok(Timestamp(Nanosecond, None)),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matching the old behaviour wrt nulls here

Ok(ColumnarValue::Array(Arc::new(builder.finish())))
}

fn to_local_time(time_value: &ColumnarValue) -> Result<ColumnarValue> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored to_local_time function here now; I also flattened the function since it was quite nested before


# invalid argument data type
statement error The to_local_time function can only accept Timestamp as the arg got Utf8
statement error DataFusion error: Error during planning: Internal error: Expect TypeSignatureClass::Timestamp but received NativeType::String, DataType: Utf8.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only actual change to this file (rest is whitespace changes).

Since we're using signature API now the default error changes; technically it is less friendly than what we had before, but it means it's centralized and we can work towards ensuring the message is better for the whole signature coercion API

@Jefffrey
Jefffrey marked this pull request as ready for review November 14, 2025 16:10
@codetyri0n

Copy link
Copy Markdown
Contributor

reckon SET datafusion.execution.time_zone = ' '; would make it non deterministic, so lgtm

Some(adjusted_ts),
None,
)))
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about ColumnarValue::Scalar(ScalarValue::TimestampXyz(None, Some(_))) => { ?
IMO it should return:

return Ok(ColumnarValue::Scalar(ScalarValue::TimestampNanosecond(
                None, None,
            )));

as for the DateType::Null case above

@Jefffrey Jefffrey Nov 19, 2025 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, added arms and a test: 5de389e

// if no timezone specified, just return the input
return Ok(time_value.clone());
}
DataType::Null => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this case possible ?
Coercion::new_exact(TypeSignatureClass::Timestamp) should make it impossible, no ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is a small quirk of the coercion API, where Null types are passed through as is; I plan to look into it later to see if we can handle it better (need to check if there is an existing GitHub issue for this)

) -> Result<ColumnarValue> {
let mut builder = PrimitiveBuilder::<T>::new();

let primitive_array = as_primitive_array::<T>(array)?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let primitive_array = as_primitive_array::<T>(array)?;
let primitive_array = as_primitive_array::<T>(array)?;
let mut builder = PrimitiveBuilder::<T>::with_capacity(primitive_array.len());

move the builder declaration here to have the primitive_array size.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added in e547c5a

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍


# invalid argument data type
statement error The to_local_time function can only accept Timestamp as the arg got Utf8
statement error DataFusion error: Error during planning: Internal error: Expect TypeSignatureClass::Timestamp but received NativeType::String, DataType: Utf8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems wrong to me that a user error on query returns an internal error -- it should be a planning error I would think

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intend to look into cleaning up the error messages from using the coercion signature api soon 👍

@alamb

alamb commented Nov 25, 2025

Copy link
Copy Markdown
Contributor

Thanks again @Jefffrey and @martin-g for making this code better

@Jefffrey
Jefffrey added this pull request to the merge queue Nov 26, 2025
Merged via the queue into apache:main with commit 473cbda Nov 26, 2025
@Jefffrey
Jefffrey deleted the refactor-to-local-time-sig branch November 26, 2025 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants